Skip to content

Add element_type/name_pattern/level filters to get_elements, get_element_names, get_elements_dataframe#1401

Merged
MariusWirtz merged 4 commits into
cubewise-code:masterfrom
meyersrl:feat/element-filtering-on-get-elements
Jun 23, 2026
Merged

Add element_type/name_pattern/level filters to get_elements, get_element_names, get_elements_dataframe#1401
MariusWirtz merged 4 commits into
cubewise-code:masterfrom
meyersrl:feat/element-filtering-on-get-elements

Conversation

@meyersrl

Copy link
Copy Markdown
Contributor

Summary

Adds three optional, AND-composable filter kwargs — element_type, name_pattern, level — to get_elements, get_element_names, and get_elements_dataframe. Translates to OData $filter server-side. Refactors get_elements_by_level and get_elements_filtered_by_wildcard to delegate to the new path (behavior preserved).

Motivation

Today TM1py can count elements by type (get_number_of_leaf_elements, get_number_of_consolidated_elements, etc.) but can't list them without pulling the entire dimension and filtering in Python, or hand-writing MDX. This closes the gap with a small, additive API.

# Before — pull everything, filter in Python:
all_elements = tm1.elements.get_elements(dim, hier)
numeric_leaves = [e for e in all_elements if e.element_type == Element.Types.NUMERIC]

# After:
numeric_leaves = tm1.elements.get_elements(dim, hier, element_type="numeric")

# Or compose all three:
tm1.elements.get_element_names(
    dim, hier,
    element_type=["numeric", "consolidated"],
    name_pattern="Region*",
    level=1,
)

API

element_type accepts Element.Types enum, str ('numeric'/'string'/'consolidated', case-insensitive), int (1/2/3), or an iterable of any of those (OR-combined).

name_pattern is a glob with * wildcard, case- and space-insensitive (matching the semantics of the existing get_elements_filtered_by_wildcard). Translates to startswith, endswith, contains, or eq depending on * placement. ? raises ValueError.

level is exact match on the hierarchy level (0 = leaf).

All three default to None — purely additive, no breaking changes.

Design notes worth flagging

1. Incidental fix in get_elements URL. The original get_elements URL used ?select=Name,Type (missing the $ prefix), which TM1's OData endpoint silently ignored — the response always carried the full element payload. This PR corrects it to ?$select=Name,Type. Element.from_dict handles UniqueName, Index, and Attributes via .get(..., None) so no existing caller breaks. Same latent bug exists at get_edges (line 706) but is out of scope for this PR.

2. element_type overrides skip_consolidations. In get_elements_dataframe, when element_type is explicitly set alongside skip_consolidations=True, element_type is authoritative. The two kwargs are not AND-combined — this is a deliberate design choice documented in the docstring. (Setting element_type=['numeric','consolidated'] while leaving skip_consolidations=True would otherwise be incoherent: you asked for consolidations but the default flag drops them.)

3. Empty-match dataframe schema. When the trio filter matches zero elements, get_elements_dataframe constructs an empty MDX set via Tm1FilterByLevel({Members}, 9999) rather than the bare {}. The bare empty set loses the dimension column in the downstream pd.merge, breaking the DataFrame schema for callers that index by attribute or level column. The 9999-level filter produces an empty result while preserving the full column schema. A regression test (test_dataframe_trio_empty_match_preserves_schema) guards this.

Test plan

  • ~48 pure-function unit tests for the two new private helpers (_coerce_element_types, _build_elements_filter) in Tests/ElementService_filtering_helpers_test.py, covering translation, type coercion, glob-to-OData conversion, single-quote escaping, and every validation error path. No TM1 connection required.
  • ~37 integration tests in Tests/ElementService_test.py::TestElementFiltering against a fixture dimension covering each kwarg alone, in combination, with case/space variations, quote-escaped names (O'Brien), and validation passthrough.
  • Regression tests confirming get_elements_by_level and get_elements_filtered_by_wildcard produce identical output to master (snapshots committed under Tests/fixtures/element_filtering_snapshots/).
  • Regression test confirming get_elements_dataframe produces an identical DataFrame to master when no trio kwargs are passed.

Total ~85 new tests. Verified against a live TM1 11.8.x server: all green, no regressions in the existing 115 TestElementService tests.

Backwards compatibility

Purely additive. All new kwargs default to None. No signatures removed, no return types changed, no error semantics changed for any existing path.

Files touched

  • TM1py/Services/ElementService.py — two new private helpers, kwargs added to three public methods, two existing methods refactored to thin delegations.
  • Tests/ElementService_filtering_helpers_test.py — new file, unit tests for the helpers.
  • Tests/ElementService_test.py — new TestElementFiltering class.
  • Tests/fixtures/element_filtering_snapshots/ — regression oracle snapshots (9 files).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds optional, AND-composable server-side OData filtering (element_type, name_pattern, level) to element-listing APIs in ElementService, and refactors existing helper methods to delegate to the new filtering path. This closes a gap where callers previously had to fetch all elements and filter client-side (or write MDX).

Changes:

  • Add element_type / name_pattern / level kwargs to get_elements, get_element_names, and get_elements_dataframe, translating to OData $filter.
  • Refactor get_elements_by_level and get_elements_filtered_by_wildcard to delegate to get_element_names.
  • Add unit + integration tests and regression snapshots covering coercion, filter generation, and dataframe schema behavior.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
TM1py/Services/ElementService.py Implements the new trio-filter API, OData filter construction helpers, and refactors delegation paths
Tests/ElementService_test.py Adds integration/regression tests for the new filtering behavior and dataframe schema guarantees
Tests/ElementService_filtering_helpers_test.py Adds pure unit tests for element-type coercion and OData filter generation
Tests/fixtures/element_filtering_snapshots/wildcard_0.json Snapshot fixture for wildcard regression
Tests/fixtures/element_filtering_snapshots/wildcard_1.json Snapshot fixture for wildcard regression
Tests/fixtures/element_filtering_snapshots/wildcard_2.json Snapshot fixture for wildcard regression
Tests/fixtures/element_filtering_snapshots/wildcard_3.json Snapshot fixture for wildcard regression
Tests/fixtures/element_filtering_snapshots/wildcard_4.json Snapshot fixture for wildcard regression
Tests/fixtures/element_filtering_snapshots/by_level_0.json Snapshot fixture for by-level regression
Tests/fixtures/element_filtering_snapshots/by_level_1.json Snapshot fixture for by-level regression
Tests/fixtures/element_filtering_snapshots/by_level_2.json Snapshot fixture for by-level regression
Tests/fixtures/element_filtering_snapshots/dataframe_default.csv Snapshot fixture for dataframe regression

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread TM1py/Services/ElementService.py
Comment thread TM1py/Services/ElementService.py Outdated
@MariusWirtz

Copy link
Copy Markdown
Collaborator

Hi @meyersrl,

Sorry for the late review. This looks good and useful! Thanks for the addition.

We have a small conflict now because other PRs have been merged in the meantime, and Copilot left a few comments.
I think we can merge after we resolve these two issues

…element_types)

Adds two private module-level helpers in ElementService.py that translate
optional element_type / name_pattern / level filters into OData $filter
clauses. No public API change yet — subsequent commits wire these into
get_elements, get_element_names, and get_elements_dataframe.

Includes ~50 pure-function unit tests covering translation, type coercion,
glob-to-OData conversion (startswith/endswith/contains/multi-segment),
OData single-quote escaping, and all validation error paths.
…d get_element_names

Adds three optional, AND-composable filter kwargs to the two list-returning
element retrieval methods. Filters translate to OData $filter server-side
via _build_elements_filter (introduced in the previous commit).

element_type accepts Element.Types enum, str ('numeric'/'string'/
'consolidated', case-insensitive), int (1/2/3), or an iterable of those
(OR-combined). name_pattern is a glob with '*' wildcard, case- and
space-insensitive (matching the semantics of get_elements_filtered_by_wildcard).
level is an exact match on the hierarchy level integer.

All three default to None, purely additive, no breaking changes. Includes
integration tests against a fixture dimension covering each kwarg alone,
in combination, with case/space variations, quote-escape, and validation
error paths.
… trio kwargs to get_elements_dataframe

get_elements_by_level and get_elements_filtered_by_wildcard are now thin
delegations to get_element_names with the appropriate kwargs. Behavior is
preserved (verified by regression tests against snapshots captured from
master before the refactor). Net effect: single source of truth for OData
$filter construction across the four element-listing methods.

get_elements_dataframe gains element_type / name_pattern / level kwargs.
When any of the trio is set while elements is None, the method resolves the
selection via get_element_names and feeds it into the existing MDX path.
The trio is authoritative and overrides skip_consolidations (documented in
the docstring).
- _odata_str_literal now percent-encodes URL-reserved characters (%, #, ?, &)
  by delegating to build_url_friendly_object_name. Previously a name_pattern
  like 'Sales & Marketing' would corrupt the URL query string at &$filter=.

- get_elements_dataframe trio-filter path now forwards **kwargs to
  get_element_names so request-level options (timeout, cancel_at_timeout,
  async_requests_mode) reach the REST call, matching the rest of the method.

Added unit tests covering both behaviors.
@meyersrl meyersrl force-pushed the feat/element-filtering-on-get-elements branch from 4b0927d to 1f0e491 Compare June 23, 2026 15:12
@meyersrl

Copy link
Copy Markdown
Contributor Author

Hi @MariusWirtz — thanks for the review! Both Copilot findings are addressed in 1f0e491, and the branch is rebased on the latest master so the conflict is resolved.

1. **kwargs forwarding in get_elements_dataframe trio path
The internal get_element_names call now forwards **kwargs so request-level options (timeout, cancel_at_timeout, async_requests_mode) reach the REST call, matching the rest of the method.

2. URL escaping in _odata_str_literal
The filter clause is concatenated raw into the URL query string (url += "&$filter=" + clause), so a name_pattern like Sales & Marketing would split the query at &. Fixed by delegating to build_url_friendly_object_name from TM1py.Utils, which is the established convention elsewhere in the codebase for user-supplied strings landing in URLs. It percent-encodes %, #, ?, & and still handles the OData single-quote doubling.

Tests added in Tests/ElementService_filtering_helpers_test.py:

  • TestOdataStrLiteralUrlSafety — 6 cases covering each reserved char individually, the OData quote-escape regression guard, and an end-to-end check via _build_elements_filter proving the produced clause is URL-safe.
  • TestGetElementsDataFrameForwardsKwargs — mocks get_element_names on a service instance and confirms timeout / cancel_at_timeout / async_requests_mode propagate.

All 55 helpers tests pass locally.

@MariusWirtz MariusWirtz merged commit 3fad65c into cubewise-code:master Jun 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants